Skip to content

fix(server): Codex runs sharing a session no longer fall behind together - #39136

Open
navidnadali wants to merge 2 commits into
pingdotgg:mainfrom
navidnadali:up/shared-session-routes-before-commit
Open

navidnadali wants to merge 2 commits into
pingdotgg:mainfrom
navidnadali:up/shared-session-routes-before-commit

Conversation

@navidnadali

Copy link
Copy Markdown
Contributor

Note

Written by Claude Opus 5.5 on behalf of Navid

Problem

Codex runs every thread of a provider instance on one provider session, so every live Codex run receives every thread's
events. A run dropped another thread's event only when it processed it (routeProviderEvent), after
consumeProviderEvents had handled it: another thread's snapshots were queued as jobs in the shared commit, and another
thread's usage report waited for those commits. With a root thread and a few delegated children running, each run
waited for every thread's commits, so on a busy writer they all fell behind together and the lag kept growing. This is
still the case after #38544. Details in #39135.

Change

RunExecutionService now routes the subscription's events as they arrive, on its own copy of the routing state, and
drops before the commit queue the ones routing rejects without changing that state. routeProviderEvent is a pure
state machine and only it sets the run's routing state, so the run would reject exactly those events when processing
them: it still accepts the same events in the same order. Before the root turn ends, a rejected event writes nothing and
its stop check returns false; past the root turn every event still reaches the run, so a run waiting on background work
rechecks as before.

Scope

Fixes #39135.

Verification

  • vp test run src/orchestration-v2/RunExecutionService.test.ts -t "keeps every run on a shared provider session"
    (from apps/server): nine runs through the real RunExecutionService on one shared ProviderCommitBatch; the writer
    is a FIFO one-permit lock that ten other writers keep busy (~1 s per commit); every 10 s each thread reports usage,
    writes two turn items and reports usage again. It requires every run to be at most 2 rounds behind after 5 minutes. On
    main it fails by assertion (6 rounds behind); with this change it passes.

  • Runtime transcript: the same model with 1-12 runs for 10 minutes, on main and on this branch (rounds behind; commits
    and queued snapshot jobs after 10 minutes):

    runs rounds behind at 1 / 5 / 10 min, main this PR commits / snapshot jobs in 10 min, main this PR
    1 0 / 0 / 0 0 / 0 / 0 59 / 118 59 / 118
    5 0 / 0 / 0 0 / 0 / 0 472 / 2,950 118 / 590
    9 0-1 / 4-5 / 9-10 0 / 0 / 0 590 / 8,022 118 / 1,062
    12 1-2 / 8-9 / 18 0 / 0 / 0 596 / 11,830 118 / 1,416

    On main the snapshot jobs grow with the square of the runs and every run waits for one commit per thread each
    round; with this change a run queues only its own two snapshots per round. The output is identical on Linux x86_64
    (Intel Xeon, Debian 13, Node 26.13.1) and macOS (Apple M5 Max). To reproduce from a checkout of this PR with
    dependencies installed: bash reproduce.sh <checkout> from https://gist.github.com/navidnadali/6571419c289985057c67f09a8adc9f91 (it appends the model to
    RunExecutionService.test.ts, runs it with main's and this PR's RunExecutionService.ts, prints the table and
    restores both files).

  • vp run check:changed passes; vp test run src/orchestration-v2 (with the provider replay fixtures) 3,152/3,152 (124 files);
    vp test run integration from apps/server (all 26 *.integration.test.ts files) 394/394.

Not checked: a live Codex session; the model stands in for the writer load.

Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.

🤖 Generated with Claude Code

Codex runs every thread of an instance on one provider session, so each
run receives every thread's events. A run dropped another thread's events
only when it processed them: their snapshots were queued as commit jobs
and their usage reports waited for those commits, so with several runs
live each one waited for every thread's commits and all fell behind.

A run now routes its events as they arrive and drops, before the commit
queue, the ones it would reject without a routing change. Past the root
turn every event still reaches the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Nov 19, 2026
@juliusmarminge juliusmarminge added the takeover:in-progress A takeover lead is working on this PR label Nov 19, 2026
…tes-before-commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ef47de4 and a00b167.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The provider-event stream now applies routing before commit queuing and filters certain events before the root turn ends. A shared-session test exercises event ingestion while competing writers hold a semaphore.

Changes: Provider event routing

Layer / File(s) Summary
Route events before commit queuing
apps/server/src/orchestration-v2/RunExecutionService.ts
Before the root turn ends, the stream filters events that are neither accepted nor routing-state updates. After it ends, the stream passes all events downstream.
Test shared-session ingestion under contention
apps/server/src/orchestration-v2/RunExecutionService.test.ts
The background scenario can use a supplied shared commit coordinator. A new test runs nine consumers with ten competing writers and asserts that maximum ingestion lag stays within two rounds after five simulated minutes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: shivamhwp, juliusmarminge, t3dotgg

Merge Risk: Minimal · previous review ef47d

This change stops runs that share a provider session from queuing work for events they will reject, so they no longer fall behind together under writer load. The supplied evidence shows no concrete regression, and the new test covers the contention scenario. Merge risk is low.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: Codex runs that share a provider session no longer fall behind together. It uses a conventional commit prefix and is specific. The changeset routes provider events ahe…
Description check Passed The description follows the template. It has Problem, Change, Scope, and Verification sections. The Problem and Change sections explain the lag cause and the routing fix. Scope links #39135. Verificat…
Linked Issues check Passed Directly linked issue #39135 asks that runs sharing a provider session drop other threads' events without waiting on the shared writer. The PR summary says routedAhead in RunExecutionService.ts ap…
Out of Scope Changes check Passed The changes touch two files: RunExecutionService.ts (about +11 lines, the pre-queue routing step) and RunExecutionService.test.ts (about +95/-2, the shared-coordinator test support and the new reg…
Full details: Title check

Explanation

The title names the main change: Codex runs that share a provider session no longer fall behind together. It uses a conventional commit prefix and is specific. The changeset routes provider events ahead of the commit queue and drops rejected events, which addresses the lag described in the title.

Full details: Description check

Explanation

The description follows the template. It has Problem, Change, Scope, and Verification sections. The Problem and Change sections explain the lag cause and the routing fix. Scope links #39135. Verification gives focused test commands, a benchmark table, and the full suite results, and it states what was not checked, a live Codex session. The author also notes the model used, as the template asks. The Scope section links the issue but does not explain a reason for a bug fix that lacks an issue, which is not needed because an issue exists. The description is thorough and complete.

Full details: Linked Issues check

Explanation

Directly linked issue #39135 asks that runs sharing a provider session drop other threads' events without waiting on the shared writer. The PR summary says routedAhead in RunExecutionService.ts applies routeProviderEvent before the commit queue and drops events that routing rejects without changing routing state. Before the root turn ends, rejected events write nothing. After the root turn ends, all events still pass through, so background-work checks continue. The reviewed head confirms that routeProviderEvent is a pure function over ProviderEventRoutingState that returns [boolean, state], which supports the claim that the pre-queue check matches the in-run check. The test keeps every run on a shared provider session in RunExecutionService.test.ts models nine runs on one shared ProviderCommitBatch with a busy writer and asserts at most two rounds of lag after five minutes. The PR reports that this test fails on main (6 rounds behind) and passes with the change. The issue's Codex-only framing is broader in the code: the fix sits in the generic routing path, so it covers any provider with supportsMultipleProviderThreadsPerSession. No acceptance criteria beyond the reported behavior are specified. Limits: the diff hunks for RunExecutionService.ts could not be read (the diff tool reported missing blobs, and the full-file read was truncated), so the routedAhead logic rests on the PR summary and the test description. Test results are author-reported and were not re-run.

Full details: Out of Scope Changes check

Explanation

The changes touch two files: RunExecutionService.ts (about +11 lines, the pre-queue routing step) and RunExecutionService.test.ts (about +95/-2, the shared-coordinator test support and the new regression test). Both serve the objective in #39135: the source change implements the routing, and the test checks the lag behavior. The test support adds Effect Context and Semaphore imports, which the new test needs. No unrelated files or API declarations change (the summary reports no changes to exported declarations). The PR description's verification runs (check:changed, the orchestration-v2 and integration suites) are validation of this change, not separate scope. Limit: the exact RunExecutionService.ts hunks could not be read, so this verdict relies on the change summary.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0317d4f and ef47de4.

Files selected for processing: 2

  • apps/server/src/orchestration-v2/RunExecutionService.test.ts
  • apps/server/src/orchestration-v2/RunExecutionService.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@juliusmarminge

Copy link
Copy Markdown
Member

The automated pipeline stopped on this pull request and will not merge it:

the capture step could not show the problem on the base commit, so there is nothing to prove fixed.

Read the scenario guidance, which excludes timing and provider-specific performance proof. Started the head app and completed a real Codex turn. Requested eight concurrent native children from one parent. Only three launched. Configured provider launch arguments with agents.max_threads=12 and repeated; Codex again reported three successful launches and five errors: collab spawn failed: agent thread limit reached. Uploaded the scripted diagnostic. Stopped the server and removed the worktree and scratch files. The base app was not run.

It stays open for a person to finish or close.

@juliusmarminge juliusmarminge added takeover:blocked Takeover blocked; reason in the PR body and removed takeover:in-progress A takeover lead is working on this PR labels Nov 19, 2026
@navidnadali

Copy link
Copy Markdown
Contributor Author

Note

Written by Claude Opus 5.5 on behalf of Navid

We ran this shape live with Codex: twelve T3 threads on one Codex provider instance (so one shared provider
session), all started at once, each asked for ten rounds of "stream a 100-line numbered list, then run a
checkpoint command" (gpt-6-luna, effort low, a fresh database per run, one server at a time; macOS arm64,
Node 26.13.1, codex-cli 0.169.0). Codex's own
item/completed frames from the provider logs were compared with read-only database samples every 250 ms.

main f03ce68 this PR a00b167
Threads streaming a list at once (peak) 10 10
When the last of those lists finished in Codex: lines Codex had finished / lines stored 1,000 / 1,000 1,000 / 1,000
Rounds behind (Codex's completed checkpoints minus stored), every 15 s 0 0
Shared-commit jobs over the turns (per stored list line) 41,228 (34.4) 5,936 (4.2)
Commit batch wait p90 / max 15 / 63 ms 4 / 38 ms

So this live run does not reproduce the lag on main. Every run did queue every thread's snapshots (about
eight times this PR's commit jobs for the same output), but with no other writer a commit waited milliseconds,
so no thread fell behind. The lag needs commits that wait about a second, as in the issue's model, and a fresh
dev database does not get there. The model also ended most turns after the first list, so the threads
streamed together for about 15 s rather than minutes. Driver, analysis and steps: https://gist.github.com/navidnadali/18d79e2ae998c645a43f1904c7d3243d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). takeover:blocked Takeover blocked; reason in the PR body vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Codex runs that share a provider session fall behind together while the writer is busy

2 participants